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ABSTRACT 

Previously, we showed that air traffic controllers (ATCos) 
rated UAS pilot verbal response latencies as acceptable 
when a 1 .5 s delay was added to the UAS pilot responses, 
but a 5 s delay was rated as mostly unacceptable. In the 
present study we determined whether a 1.5 s added delay in 
the UAS pilots’ verbal communications would affect 
ATCos interactions with UAS and other conventional 
aircraft when the number and speed of the UAS were 
manipulated. Eight radar-certified ATCos participated in 
this simulation. The ATCos managed a medium altitude 
sector containing arrival aircraft, en route aircraft, and one 
to four UAS. The UAS were conducting a surveillance 
mission and flew at either a “slow” or “fast” speed. We 
measured both UAS and conventional pilots’ verbal 
communication latencies, and obtained ATCos’ 
acceptability ratings for these latencies. Although the UAS 
pilot response latencies were longer than those of 
conventional pilots, the ATCos rated UAS pilot verbal 
communication latencies to be as acceptable as those of 
conventional pilots. Because the overall traffic load within 
the sector was held constant, ATCos only performed 
slightly worse when multiple UAS were in their sector 
compared to when only one UAS was in the sector. 
Implications of these findings for UAS integration in the 
NAS are discussed. 
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INTRODUCTION 

Unmanned Aerial Systems (UAS) have a variety of 
practical uses such as land surveying, disaster response 
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assistance, law enforcement, and climate research. For 
example, in 2006 NASA’s Ikhana UAS was employed to 
assist fire-fighting efforts by providing timely information 
on the progression of wildfires and their hot spots [1]. The 
recognized value of UAS has led to the passage of the FAA 
(Federal Aviation Administration) Modernization and 
Reform Act of 2012, which calls for standards for UAS 
certification and integration in the NAS (National Airspace 
System). 

Currently, UAS may obtain approval from the FAA to 
operate within the NAS by either being granted an 
experimental airworthiness certificate for private sector 
operations, or a Certificate of Authorization (COA) for 
public sector operations. Neither of these methods, though, 
allows for UAS to have routine, unrestricted access to the 
NAS. For UAS to be integrated into the NAS, the “UAS 
will be required to act and respond as manned aircraft do” 
(p. 5) [2]. This requirement involves not only assessing the 
performance capabilities of the UAS, but also its 
communication characteristics when interacting with air 
traffic controllers (ATCos) [3]. 

In terms of communication with ATCos, although FAA 
regulations require pilots of conventional aircraft to 
respond promptly, they do not specify a precise time limit 
on communication delays or pilot execution of ATCo 
commands. If these guidelines are transferred to UAS 
operations, prompt responding needs to be further specified 
because UAS differ from manned aircraft in many respects 

[4] , and these differences can have an impact on the 
timeliness of the UAS responses. For example, UAS pilots 
are not co-located with their aircraft and are required to 
interact with their aircraft using ground control stations that 
currently have complex control interfaces [1]. They lack 
many sensory cues and can only rely on their instrument 
indicators and cameras with limited spatial resolution and 
small fields of view. As a result, UAS pilots have less 
information about their aircraft’s surrounding environment 

[5] . Moreover, UAS communication time can be affected 
by systemic aspects, such as whether the UAS is within the 
line of sight or communicating via satellite. Thus, UAS 
pilots may take longer than conventional pilots to 


determine whether they can safely carry out a command 
issued by an ATCo. 

For conventional aircraft, pilot delays in verbal responding 
have been shown to negatively affect the performance of 
ATCos [6]. As a result, ATCos may adjust their 
communication style in response to pilot verbal delays by, 
for example, increasing the complexity of commands. 
However, this can result in more communication errors [7]. 
For UAS, two studies have examined the effect of 
simulated UAS pilot delays on ATCo acceptability ratings 
of their response time [3], as well as the impact of 
additional delays on ATCo performance [8]. 

For one, Shivley et al. [3] examined the pilot-ATCo 
interaction in terms of a “measured response” (MR). The 
MR reflects several time components from the end of the 
ATCo’s command transmission to the first noticeable 
display of the aircraft executing the instruction on the 
ATCo’s radar scope. The first MR component (MR1) is 
measured from the end of the controller’s instruction to the 
beginning of the pilot’s verbal response. MR2 is from the 
end of the controller’s instruction to when the pilot begins a 
control input on the ground control station. MR3 is the 
interval between the end of the pilot’s control input to the 
onset of the aircraft’s execution of the maneuver. MR4 is 
the time from the initiation of the aircraft’s execution of the 
maneuver to the maneuver appearing on the ATCo radar 
scope (see Figure 1). As shown in Figure 1, these MR 
components are affected by many factors. For the purpose 
of the present study we focus only on the first component, 
MR 1: Pilot verbal communication latency. 

^ MR2: Pilot Execution Initiation L a tency ^ MR4: Display Delay Latency 
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Figure 1. End-to-end response time for completion of ATC 
clearances. Figure based on Shively et al. (20T3). 

Shively et al. [3] had ATCos issue common commands to 
IFR-rated pilots who were trained to fly the UAS on a 
MUSIM (Multiple UAS simulator [9]) ground control 
station. The commands issued were route amendments, 
traffic calls, altitude changes, and radio frequency changes. 
The UAS pilots’ verbal delay in responding to controllers 
averaged 2.5 seconds, a response time that was rated 
acceptable by ATCos. However, the acceptability ratings 
obtained in this simulation are not likely to reflect UAS 
operations in the NAS because both the pilots and ATCos 
were only issuing and responding to individual clearances 
in serial order. That is, the ATCos were not managing any 
other traffic, and the pilots were not performing any other 
tasks. ATCo acceptability of the MR1 component will 


likely differ in the presence of other traffic and with the 
addition of UAS communication delays. 

Vu et al. [8] examined how short (1.5 sec) and long (5 sec) 
delays added to the pilot verbal latencies (MR1) and 
execution initiation latencies (MR2) influenced ATCo 
performance and their acceptability ratings of the UAS 
pilots relative to conventional pilots in a simulated NAS 
environment. They found that the ATCos rated UAS pilot 
verbal communication latencies to be more acceptable 
when the inserted delays were short (1.5 s) rather than long 
(5 s); however, the acceptability ratings were based on 
other features of the sectors being managed. For example, 
the ATCos rated the acceptability of UAS delays within a 
scenario based on all aircraft rather than the UAS alone, 
because differences between the acceptability ratings of 
UAS compared with conventional aircraft, were minimal, 
even though delays were added only to the UAS responses. 
Moreover, the ATCo acceptability ratings were correlated 
with measures of their performance in a scenario, for 
example, the number of losses of separation (LOS), number 
of step-ons that occurred, and the efficiency of their traffic 
management in the scenario was worse with long UAS 
delays. 

The present study extends the work of Shively et al. [3] and 
Vu et al. [8] by examining the impact of multiple UAS in a 
simulated NAS environment. Only a short delay of 1.5 s 
was added to UAS pilot verbal communications since this 
value was found to be acceptable to ATCos. We 
manipulated both the number of UAS in the sector, (one, 
two or four), and their speeds. UAS flew at either “slow” 
speeds of 120 kts (representing the characteristics of a 
Predator) or “fast” speeds of 240 kts (representing the 
characteristics of a Global Hawk). We examined ATCo 
performance and acceptability ratings of UAS pilots’ and 
conventional pilots’ verbal response latencies (MR1) as a 
function of the number of UAS and the UAS speed in the 
sector. The data reported in this paper reflects only part of 
the data collected from the larger simulation. 

METHOD 

Participants 

Eight radar-certified ATCos vohinteered to participate in 
this simulation. The ATCos averaged over 25 years of 
military and civilian experience in air traffic management. 
All had prior experience with ZLA airspace during their 
active air traffic management period. The simulation lasted 
two days and participants were compensated $60 per hour 
for their time. 

Design 

This simulation followed a 3 (Number of UAS: 1, 2, or 4) x 
3 (Speed: Slow, Mixed, or Fast) within-subjects, repeated 
measure design (see Figure 2). 1 


1 As noted earlier in the paper, only a portion of the data 
from the full simulation is being reported here. 
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Figure 2. Illustration of the 3 (No. of UAS) 
design 

x 3 (Speed) 


Simulation Environment 

The simulation was run in three separate rooms: one room 
to manage the simulation and run the ATCo participants, a 
second room for conventional pilots, and the third for UAS 
pilots. All pilots were trained experimental confederates. 

The simulation was run using the Multi Aircraft Control 
System (MACS [10]), creating a medium-fidelity 
environment. MACS simulated a controllers DSR display 
for sector ZLA 20 (Figure 3). The multi-aircraft mode was 
used by pseudopilots to control all conventional aircraft. 
The single pilot mode was used for each of the UAS in the 
sector. The ATCo, conventional pseudopilots, and UAS 
pseudopilots spoke to each other using push-to-talk 
headsets over the voice server. The voice system was 
modified for the UAS stations. A fixed delay of 1.5 s was 
inserted before transmitting the UAS pseudopilots’ 
message to the ATCo and conventional pseudopilots. 



Two parallel worlds were run at a time. Nine to 12 
computers were required for each simulated world. The 
controller’s station included two computers, one to simulate 


their radar scope and a second small display acting as 
‘Mission Control’ to provide instructions to the controller 
regarding altitude changes to the UAS. Two stations were 
used for each conventional aircraft pseudopilot: one station 
was a “ghost” controller station that allowed them to check 
the traffic coming into the sector and the second to control 
all aircraft, with the exception of the UAS, within sector 
ZLA 20. One to four stations were used to control the UAS 
aircraft, depending on the number of UAS in the sector 
during each scenario. Additional computers were used to 
manage the simulation software, record communication 
data between the controller and pseudopilots over the voice 
server, and for the ADRS simulation hub. The voice 
software was also modified to simulate stepped-on 
transmissions so that when simultaneous transmissions 
occurred, the transmission would be unintelligible for 
everyone listening on that frequency. 

Procedure 

The simulation was conducted over two days for each 
participant. On the morning of the first day, participants 
completed consent forms and demographic questionnaires, 
and were briefed on simulation procedures. Following the 
briefing, the controllers worked three practice trials and 
nine experimental trials for the remaining time. 

Each experimental trial lasted 40 minutes. During the trial, 
controllers managed all air traffic coming into their sector. 
Controllers were instructed that arrival traffic on approach 
to LAX had priority and were required to leave the sector at 
an altitude no greater than 11,000 feet, and at an air speed 
of 250 knots. The controllers were told that their sector had 
a Letter of Agreement to accommodate requests regarding 
the UAS in order to fulfill the UAS flight objectives while 
maintaining safe operations for all air traffic. All UAS 
were given a Predator callsign (e.g., PD-1) for the “slow” 
UAS and a Global Hawk callsign (e.g., GH-1) for the “fast” 
UAS. 

Beginning approximately one to two minutes into the trial, 
and occurring every four to five minutes afterwards, either 
"Mission Control" or the UAS pseudopilot initiated a 
flight-change request to sector ATCo. Requests from 
"Mission Control," the screen to the right of the controller’s 
radar scope, alerted the controller through his headset and 
display of a new, optimal altitude that the UAS should be 
flying to accomplish their mission objective (ex. "PD-1 
climb and maintain 14000"). Mission control only 
requested altitude changes for the UAS. During the next 
period the UAS pseudopilot requested a speed change. A 
total of 8 requests for all UASs was completed each trial, 
four initiated by the controller and four by the UAS 
pseudopilot. When multiple UAS were in the sector, the 
total number of communications was divided evenly 
between each UAS. 

Voice delays were inserted at the UAS pseudopilot station. 
Voice software at this station automatically held the UAS 
pilot’s audio transmission for 1.5 seconds before 
broadcasting to the ATCo and conventional pseudopilots. 


The UAS station only included a transmission delay, not a 
receiving delay. The controller and conventional 
pseudopilots had no receiving or transmission delays. 

After each trial, controllers rated their situation awareness 
[11], workload [12], and experience interacting with the 
conventional and UAS pseudopilots. For workload, the 
Once all nine experimental trials were completed, 
controllers answered post-simulation and debriefing 
questionnaires. Finally, the controllers were interviewed 
during a session covering the same topics as the debriefing 
survey. 

RESULTS 

A series of 3 (Number of UAS: 1, 2, or 4) X 3 (Speed: 
slow, mixed, fast) repeated measures ANOVAs were 
performed on ATCo’s performance, workload, and 
acceptability ratings of UAS and conventional pilot verbal 
response delays. 

ATCo Performance 

ATCo performance was examined by the number of losses 
of separation (LOS; a safety metric) and distance it took the 
conventional AC to travel through the sector (an efficiency 
metric). Overall, the number of LOS averaged 2.6 ( SE = 
.28). Although the number of LOS was not significantly 
affected by either number of UAS or speed, we did observe 
a significant linear trend between number of UAS and 
number LOS, F{ 1,7) = 5.42; p = .05 (see Figure 4). The 
linear component accounted for 44% of the variance in 
LOS. Moreover, about half the number of LOS involved a 
UAS. 


Loss of Separation 
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Figure 4. Mean Number of Losses of Separation as a 
function of Number of UAS 


For distance traveled, there was a significant Number of 
UAS x Speed interaction, ^(4,28) = 5.026, p = .004, see 
Figure 5. For one UAS in the sector, the distance 
conventional AC travelled was not affected by UAS speed. 
However, when there were two UAS in the sector, the 
average distance conventional AC travelled through the 
sector was higher when the UAS was at slow compared to 
fast or mixed speeds. In contrast, when there were four 
UAS in the sector, the slow UAS decreased the average 
distance travelled relative to four mixed and four fast UAS. 


In short, more efficient traffic flows occurred with faster 
speeds for two UAS in the sector, and with slower speeds 
when there were four UAS in the sector. This particular 
finding may be a result of the specific traffic pattern and 
UAS flight paths used in the present study. 
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Figure 5. Number of UAS x Speed interaction for 
distance AC traveled through the sector 

Workload 

Performance-based workload was examined through the 
hand-off accept times (shorter latencies correspond to 
lower workload), and subjective workload was examined 
with NASA-TLX ratings. 

For handoff-accept time, there was a main effect of number 
of UAS, F( 2,14) = 10.536, p = .002, where the handoff 
accept time linearly decreased with the number of UAS, see 
Figure 6. The linear trend was again significant F(\,l) = 
16.1; p = .005, and was likely caused by holding the 
average number of AC in the sector relatively constant at 
roughly eight AC. Consequently, when four UAS, were in 
the sector, there were fewer conventional AC entering the 
sector. 
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Figure 6. Main effect for Hand-off Accept Time x 
Number of UAS 
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Figure 7. Number of UAS x UAS Speed interaction for 
Hand-off Accept Time 

The main effect was modified by a significant Number of 
UAS x Speed interaction, F(4,2 8) = 3.16, p = .029, see 
Figure 7. With a single UAS in the sector, the hand-off 
accept time was shorter when the UAS was slow. With two 
fast UAS, hand-off accept times were shorter than with two 
UAS traveling at slow or mixed speeds. However, with 
four UAS, constant slow or mixed speeds led to shorter 
hand-off accept times than constant fast speeds. Thus, 
workload was lower for slow UAS when there was one or 
four in the sector, but workload was higher for the slow 
UAS when there were two in the sector. 

For the NASA-TLX, ATCos rated the overall workload as 
average (M = 53). Only the main effect of Number of 
UAS, F( 2,14) = 2.730, p = .025 was significant. In contrast 
to the hand-off accept times, subjective workload was rated 
the lowest when there were two UAS in the sector (M= 49) 
compared with one (M= 56) or four (M= 55) UAS. This 
likely reflects key contextual factors of the scenarios. 
When there was only one UAS in the sector, there were 
many more aircraft entering the sector than in the other 
conditions, thereby increasing the perceived workload. On 
the other hand, when there were four UAS, although there 
were fewer aircraft entering the sector than when there 
were two UAS, there was more LOS, as shown above. 
Increasing the number of operational errors in the four 
UAS condition therefore could have affected the subjective 
assessment of the workload created by this condition, as 
ATCos more likely had to work harder to maintain safe 
separation minima. 

Acceptability Ratings 

ATCo acceptability ratings of pilot verbal delays were 
analyzed using a 3 (Number of UAS: 1, 2, or 4) X 3 
(Speed: slow, mixed, fast) X 2 (Pilot: UAS, conventional) 
repeated measures ANOVA. Overall, all conditions were 
rated acceptable (above 4.0 on a scale of 1 = not acceptable 
to 7 = very acceptable). The ratings did not differ 
significantly as a function of number or speed of the UAS, 
or as a function of pilot type (see Table 1). 


Table 1. ATCo Acceptability Ratings (1 = not 
acceptable to 7 = very acceptable) for UAS and 
Conventional Pilot Verbal Response Latencies 


Condition 
(No. UAS-Speed) 

UAS Pilots 

Conventional Pilots 

1-Slow 

5.25 

5.38 

1-Mixed 

4.88 

5.25 

1 -Fast 

4.88 

6.25 

2-Slow 

5.50 

5.63 

2-Mixed 

5.13 

5.63 

2-Fast 

5.25 

6.38 

4-Slow 

5.13 

5.75 

4-Mixed 

4.75 

6.25 

4-Fast 

5.13 

5.75 


DISCUSSION 

The present study inserted an additional 1.5 s verbal 
response delay into UAS pilot communications, a value 
that previously resulted in an acceptable rating of verbal 
response latencies [8]. However, we increased the number 
and speeds of UAS flying in the sector to determine 
whether the safety and efficiency with which ATCos 
managed traffic would be affected. We also examined 
ATCO’s acceptability ratings of the delays under these 
conditions. 

We found negative effects of the number of UAS on a 
measure of safety, the number of LOS. Although there 
were no significant differences in LOS, the linear trend 
observed indicates that safety is compromised with 
increasing numbers of LOS in the sector. Moreover, we 
found that a UAS was involved in at least half of the LOS 
occurrences, again suggesting that the increasing number of 
UAS did negatively impact the ATCos’ performance to 
some degree. In addition, the efficiency with which the 
ATCos managed traffic was affected by the number and 
speeds of the UAS. Specifically, we found an interaction 
between number of UAS and speed, such that whether there 
were two or four UAS determined if faster UAS speed was 
related to greater distance of other aircraft through the 
sector or lesser distance through the sector. This shows that 
the effect of a single variable like speed depends on other 
contextual factors. 

With respect to workload, we found that the more UAS in 
the sector, the lower the hand off accept time, an objective 
measure of workload. Prima facie this seems 
counterintuitive. However, we need to keep in mind that as 
the average number of aircraft (including UAS) in the 
sector was held relatively constant. That is, when there 
were more UAS present, there were fewer conventional 
aircraft entering the sector, and thus fewer hand-off- 




acceptances were required. This reduction in the number of 
aircraft being handed off may explain the reduction in 
hand-off accept time. Additionally, ATCos reported that 
managing the conventional aircraft required more 
interactions than managing the UAS because the latter, 
even when fast moving, were significantly slower than the 
conventional aircraft. Thus, ATCos reported generally 
moving all conventional air traffic around the UAS, 
avoiding as much as possible, interactions with the UAS. 
Researchers should be mindful of contextual factors, ATCo 
strategies, and other performance tradeoffs when evaluating 
the impacts of UAS in the NAS. 

Finally, we found that the ratings of acceptability of the 
UAS pilot verbal latencies were not affected by the number 
or speeds of the UAS. This suggests that delays in verbal 
responding that are 1.5 sec longer than normal are still 
acceptable to controllers. Interestingly, ATCos gave 
equivalent ratings to both UAS pilots and conventional 
aircraft pilots, despite the fact that the delay of 1.5 sec was 
only inserted in the former. This finding is similar to the 
results of Vu et al. [8] and shows that when controllers 
judge the acceptability of delays, they likely take into 
account other sector (e.g., traffic density) and performance 
(e.g., LOS) factors when rating acceptability of the 
measured response. 
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